{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "t09eeeR5prIJ"
   },
   "source": [
    "##### Copyright 2018 The TensorFlow Authors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "GCCk8_dHpuNf"
   },
   "outputs": [],
   "source": [
    "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "# https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xh8WkEwWpnm7"
   },
   "source": [
    "# 自动微分和梯度带"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "idv0bPeCp325"
   },
   "source": [
    "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://tensorflow.google.cn/tutorials/customization/autodiff\"><img src=\"https://tensorflow.google.cn/images/tf_logo_32px.png\" />在 TensorFlow.google.cn 上查看</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/zh-cn/tutorials/customization/autodiff.ipynb\"><img src=\"https://tensorflow.google.cn/images/colab_logo_32px.png\" />在 Google Colab 运行</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/zh-cn/tutorials/customization/autodiff.ipynb\"><img src=\"https://tensorflow.google.cn/images/GitHub-Mark-32px.png\" />在 GitHub 上查看源代码</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/zh-cn/tutorials/customization/autodiff.ipynb\"><img src=\"https://tensorflow.google.cn/images/download_logo_32px.png\" />下载此 notebook</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vDJ4XzMqodTy"
   },
   "source": [
    "在上一个教程中，我们介绍了 \"张量\"（Tensor）及其操作。本教程涉及[自动微分（automatic differentitation）](https://en.wikipedia.org/wiki/Automatic_differentiation)，它是优化机器学习模型的关键技巧之一。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GQJysDM__Qb0"
   },
   "source": [
    "## 创建  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cxzaxo6ff2y3"
   },
   "outputs": [],
   "source": [
    "from __future__ import absolute_import, division, print_function, unicode_literals\n",
    "\n",
    "!pip install tensorflow==2.0.0-beta1\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1CLWJl0QliB0"
   },
   "source": [
    "## 梯度带\n",
    "\n",
    "TensorFlow 为自动微分提供了 [tf.GradientTape](https://www.tensorflow.org/api_docs/python/tf/GradientTape) API ，根据某个函数的输入变量来计算它的导数。Tensorflow 会把 'tf.GradientTape' 上下文中执行的所有操作都记录在一个磁带上 (\"tape\")。 然后基于这个磁带和每次操作产生的导数，用反向微分法（\"reverse mode differentiation\"）来计算这些被“记录在案”的函数的导数。\n",
    "\n",
    "例如："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bAFeIE8EuVIq"
   },
   "outputs": [],
   "source": [
    "x = tf.ones((2, 2))\n",
    "\n",
    "with tf.GradientTape() as t:\n",
    "  t.watch(x)\n",
    "  y = tf.reduce_sum(x)\n",
    "  z = tf.multiply(y, y)\n",
    "\n",
    "# Derivative of z with respect to the original input tensor x\n",
    "dz_dx = t.gradient(z, x)\n",
    "for i in [0, 1]:\n",
    "  for j in [0, 1]:\n",
    "    assert dz_dx[i][j].numpy() == 8.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "N4VlqKFzzGaC"
   },
   "source": [
    "你也可以使用 `tf.GradientTape` 上下文计算过程产生的中间结果来求取导数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7XaPRAwUyYms"
   },
   "outputs": [],
   "source": [
    "x = tf.ones((2, 2))\n",
    "\n",
    "with tf.GradientTape() as t:\n",
    "  t.watch(x)\n",
    "  y = tf.reduce_sum(x)\n",
    "  z = tf.multiply(y, y)\n",
    "\n",
    "# Use the tape to compute the derivative of z with respect to the\n",
    "# intermediate value y.\n",
    "dz_dy = t.gradient(z, y)\n",
    "assert dz_dy.numpy() == 8.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ISkXuY7YzIcS"
   },
   "source": [
    "默认情况下，调用 GradientTape.gradient() 方法时， GradientTape 占用的资源会立即得到释放。通过创建一个持久的梯度带，可以计算同个函数的多个导数。这样在磁带对象被垃圾回收时，就可以多次调用 'gradient()' 方法。例如："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zZaCm3-9zVCi"
   },
   "outputs": [],
   "source": [
    "x = tf.constant(3.0)\n",
    "with tf.GradientTape(persistent=True) as t:\n",
    "  t.watch(x)\n",
    "  y = x * x\n",
    "  z = y * y\n",
    "dz_dx = t.gradient(z, x)  # 108.0 (4*x^3 at x = 3)\n",
    "dy_dx = t.gradient(y, x)  # 6.0\n",
    "del t  # Drop the reference to the tape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6kADybtQzYj4"
   },
   "source": [
    "### 记录控制流\n",
    "\n",
    "由于磁带会记录所有执行的操作，Python 控制流（如使用 if 和 while 的代码段）自然得到了处理。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9FViq92UX7P8"
   },
   "outputs": [],
   "source": [
    "def f(x, y):\n",
    "  output = 1.0\n",
    "  for i in range(y):\n",
    "    if i > 1 and i < 5:\n",
    "      output = tf.multiply(output, x)\n",
    "  return output\n",
    "\n",
    "def grad(x, y):\n",
    "  with tf.GradientTape() as t:\n",
    "    t.watch(x)\n",
    "    out = f(x, y)\n",
    "  return t.gradient(out, x)\n",
    "\n",
    "x = tf.convert_to_tensor(2.0)\n",
    "\n",
    "assert grad(x, 6).numpy() == 12.0\n",
    "assert grad(x, 5).numpy() == 12.0\n",
    "assert grad(x, 4).numpy() == 4.0\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DK05KXrAAld3"
   },
   "source": [
    "### 高阶导数\n",
    "\n",
    "在 'GradientTape' 上下文管理器中记录的操作会用于自动微分。如果导数是在上下文中计算的，导数的函数也会被记录下来。因此，同个 API 可以用于高阶导数。例如："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cPQgthZ7ugRJ"
   },
   "outputs": [],
   "source": [
    "x = tf.Variable(1.0)  # Create a Tensorflow variable initialized to 1.0\n",
    "\n",
    "with tf.GradientTape() as t:\n",
    "  with tf.GradientTape() as t2:\n",
    "    y = x * x * x\n",
    "  # Compute the gradient inside the 't' context manager\n",
    "  # which means the gradient computation is differentiable as well.\n",
    "  dy_dx = t2.gradient(y, x)\n",
    "d2y_dx2 = t.gradient(dy_dx, x)\n",
    "\n",
    "assert dy_dx.numpy() == 3.0\n",
    "assert d2y_dx2.numpy() == 6.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4U1KKzUpNl58"
   },
   "source": [
    "## 下一步\n",
    "\n",
    "本教程涉及 TensorFlow 中的导数计算。以此为基础，我们具备了足够的先修知识来构建和训练神经网络。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "j8BT7T8yf6Rv"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "autodiff.ipynb",
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
